H-6800: Improve Petrinaut Voice turn-taking and answer provenance by kostandinang · Pull Request #9512 · hashintel/hash

kostandinang · 2026-09-02T20:05:25Z

🌟 What is the purpose of this PR?

Make Voice interruption predictable and keep Voice attribution next to the interactive-tool answer it describes. Users can explicitly take the turn while the assistant prepares or speaks, and the microphone resumes only after Realtime confirms that canceled input and output have cleared.

This PR collects the independently useful UI/UX improvements discovered while testing H-6763. It does not change semantic VAD eagerness, add full duplex, or provide durable elicitation or canvas writes.

🔗 Related links

H-6800 (internal)
H-6763 (internal)
H-6763: Add temporary Brunch ask shim for Voice preview #9507 — immediate base containing the temporary brunch_ask shim

🚫 Blocked by

H-6763: Add temporary Brunch ask shim for Voice preview #9507 — this PR is stacked on its branch

🔍 What does this change?

Adds an accessible Your turn action while Voice prepares or speaks, styled consistently with the Voice dock controls.
Clears input and output audio, cancels queued or active speech without disconnecting Realtime, and waits for cancellation acknowledgements before reopening the microphone.
Ignores speech captured before the handoff while keeping the interrupted question visible and repeatable.
Lets interactive tools opt into a backwards-compatible answer-prefix slot so the Voice provenance chip appears immediately before submitted answers; other widgets retain the trailing fallback.
Updates the Brunch ask widget, Petrinaut AI Assistant documentation, package guidance, release notes, and controller/session/integration/UI tests.

🏗️ Agent notes

Imperative: Preserve the useful Voice interaction improvements exposed by H-6763 as a reviewable change independent of the temporary Brunch compatibility mechanism.

Throughline: Voice dock → composer control → host Voice control → turn controller → OpenAI Realtime cancellation acknowledgements → listening resumes. Interactive-tool rendering independently projects Voice provenance immediately before an opted-in answer.

Proof: Controller, Realtime session, host-control, preview-integration, and UI tests cover successful and failed handoffs, stale audio rejection, cancellation ordering, repeatability, inline provenance, and backwards-compatible fallback rendering.

Constraints: Semantic VAD remains at low eagerness so natural thinking pauses are not prematurely committed. The handoff does not disconnect Realtime or enable full duplex. Other interactive tools retain their existing provenance placement unless they opt into the prefix slot.

Fog-line: The durable elicitation and canvas-write architecture remains outside this PR. These interaction behaviors must survive any later replacement of the temporary parent shim.

Stop or reorient: Restack or migrate these behaviors onto the durable Voice branch before #9507 is removed.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

modifies the npm-publishable @hashintel/petrinaut library and includes its required patch changeset

📜 Does this require a change to the docs?

The changes in this PR:

require changes to docs which are made as part of this PR

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

do not affect the execution graph

⚠️ Known issues

Voice remains intentionally half-duplex; Your turn is an explicit handoff rather than simultaneous conversation.
This PR does not change semantic VAD eagerness; it remains low.
Durable elicitation and Petrinaut canvas writes remain outside this stack.

🐾 Next steps

Restack or migrate these behaviors onto the durable Voice implementation before the temporary H-6763: Add temporary Brunch ask shim for Voice preview #9507 shim is retired.

🛡 What tests cover this?

Focused Voice controller, Realtime session, preview integration, session-state, and host-control tests
Focused Petrinaut AI Assistant and interactive-tool UI tests
yarn workspace @apps/petrinaut-website lint:tsc
yarn workspace @hashintel/petrinaut lint:tsc
yarn workspace @apps/petrinaut-website lint:eslint — 0 errors; 1 pre-existing warning
yarn workspace @hashintel/petrinaut lint:eslint — 0 errors or warnings
Focused oxfmt, Markdown, architecture-doc, and git diff --check validation

❓ How to test this?

Start a Voice interview and wait for the assistant to prepare or speak a question.
Select Your turn and confirm speech stops without disconnecting.
Speak only after the handoff completes and confirm pre-handoff audio is not submitted.
Confirm Repeat question remains available for the interrupted question.
Submit a Voice answer through the Brunch ask widget and confirm the Voice chip appears immediately before the answer.

📹 Demo

Not recorded.

vercel · 2026-09-02T20:05:28Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hash	Ready	Preview	Sep 3, 2026 12:43pm UTC
petrinaut	Ready	Preview	Sep 3, 2026 12:43pm UTC
petrinaut-docs	Ready	Preview	Sep 3, 2026 12:43pm UTC

1 Skipped Deployment

Project	Deployment	Actions	Updated
hashdotdesign-tokens	Ignored	Preview	Sep 3, 2026 12:43pm UTC

cursor · 2026-09-02T20:05:35Z

PR Summary

Medium Risk
Changes real-time voice cancellation, microphone gating, and chat metadata attribution—behavioral paths that are easy to get wrong under race conditions, though coverage is extensive in session/controller/UI tests.

Overview
Adds a Your turn control in the Voice dock (and takeTurn / canTakeTurn on host session APIs) so users can interrupt assistant speech or pending delivery without ending the session. The website turn controller drives a stricter Realtime cancelOutput() that clears input/output buffers, awaits provider acknowledgements, and only then re-enables the mic while discarding utterances that started before the handoff.

Voice provenance moves from a text Voice chip to a compact waveform icon on spoken user messages. Interactive tools can opt in with supportsSubmittedOutputProvenance and submittedOutputProvenance so Brunch ask (and similar widgets) place that icon inside the submitted answer bubble; others keep the trailing fallback. Assistant message metadata now tracks voiceToolCallIds (with rollback when a tool submit fails) so multiple voice-answered tools on one turn stay attributed correctly.

Supporting refactors split Brunch/voice modules for Fast Refresh (widget, disclosure, config loader, getBrunchVoiceMode) and replace diagnostics transport refs with DiagnosticsTransportState in the AI assistant panel.

^{Reviewed by Cursor Bugbot for commit d13535d. Bugbot is set up for automated code reviews on this repo. Configure here.}

kostandinang · 2026-09-02T20:05:50Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

Amp-Thread-ID: https://ampcode.com/threads/T-01a06344-05c3-75c8-ac3a-d8a6fd3ede6c Co-authored-by: Amp <amp@ampcode.com>

Amp-Thread-ID: https://ampcode.com/threads/T-01a06357-5d4a-728b-8b49-57d6d1458733 Co-authored-by: Amp <amp@ampcode.com>

Amp-Thread-ID: https://ampcode.com/threads/T-01a06344-05c3-75c8-ac3a-d8a6fd3ede6c Co-authored-by: Amp <amp@ampcode.com>

Keep the controller paused until Realtime confirms its input and output buffers are clear, preventing resumed speech from being discarded. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d13535d. Configure here.}

cursor · 2026-09-03T12:35:54Z

+                      (toolCallId) => toolCallId !== params.toolCallId,
+                    );
+                if (remainingVoiceToolCallIds.length === 0) {
+                  return { ...message, metadata: previousMetadata };


Rollback restores stale voice IDs

Medium Severity

When overlapping Voice tool submissions both reject, the later rollback can restore previousMetadata after sibling IDs were already removed. That puts a failed toolCallId back into voiceToolCallIds, so a later typed retry can still receive Voice provenance.

^{Reviewed by Cursor Bugbot for commit d13535d. Configure here.}

kostandinang mentioned this pull request Sep 2, 2026

H-6763: Add temporary Brunch ask shim for Voice preview #9507

Open

4 tasks

github-actions Bot added area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) type/eng > frontend Owned by the @frontend team area/apps labels Sep 2, 2026

graphite-app Bot assigned kostandinang Sep 2, 2026

kostandinang changed the title ~~Add manual Voice turn handoff~~ H-6800: Improve Petrinaut Voice turn-taking and answer provenance Sep 2, 2026

vercel Bot deployed to Preview – petrinaut-docs September 2, 2026 20:09 View deployment

vercel Bot deployed to Preview – petrinaut September 2, 2026 20:09 View deployment

kostandinang changed the base branch from kah-6763-temporary-brunch-ask to graphite-base/9512 September 2, 2026 20:12

vercel Bot deployed to Preview – hash September 2, 2026 20:15 View deployment

cursor Bot reviewed Sep 2, 2026

View reviewed changes

Comment thread apps/petrinaut-website/src/main/app/voice-interview/voice-turn-controller.ts

kostandinang and others added 6 commits September 2, 2026 22:26

Add manual Voice turn handoff

5f66ac3

Amp-Thread-ID: https://ampcode.com/threads/T-01a06344-05c3-75c8-ac3a-d8a6fd3ede6c Co-authored-by: Amp <amp@ampcode.com>

Place Voice provenance before interactive answers

915dff2

Document manual Voice turn handoff

a6cfd99

Amp-Thread-ID: https://ampcode.com/threads/T-01a06344-05c3-75c8-ac3a-d8a6fd3ede6c Co-authored-by: Amp <amp@ampcode.com>

Consolidate Petrinaut release notes

a6eb1e6

Amp-Thread-ID: https://ampcode.com/threads/T-01a06357-5d4a-728b-8b49-57d6d1458733 Co-authored-by: Amp <amp@ampcode.com>

Match Your turn to Voice dock controls

a573816

Amp-Thread-ID: https://ampcode.com/threads/T-01a06344-05c3-75c8-ac3a-d8a6fd3ede6c Co-authored-by: Amp <amp@ampcode.com>

Wait for Voice cancellation before resuming

4422bfc

Keep the controller paused until Realtime confirms its input and output buffers are clear, preventing resumed speech from being discarded. Co-authored-by: Cursor <cursoragent@cursor.com>

kostandinang force-pushed the kah-6800-improve-petrinaut-voice-turn-taking-and-answer-provenance branch from 7d768c7 to 4422bfc Compare September 2, 2026 21:07

kostandinang requested a review from a team as a code owner September 2, 2026 21:07

kostandinang changed the base branch from graphite-base/9512 to kah-6763-temporary-brunch-ask September 2, 2026 21:07

vercel Bot deployed to Preview – petrinaut-docs September 2, 2026 21:10 View deployment

vercel Bot deployed to Preview – petrinaut September 2, 2026 21:10 View deployment

vercel Bot deployed to Preview – hash September 2, 2026 21:17 View deployment

cursor Bot reviewed Sep 2, 2026

View reviewed changes

Comment thread apps/petrinaut-website/src/main/app/voice-interview/voice-turn-controller.ts Outdated

Refine Voice answer provenance and refresh boundaries

a7b1115

Co-authored-by: Cursor <cursoragent@cursor.com>

vercel Bot deployed to Preview – petrinaut September 2, 2026 23:57 View deployment

vercel Bot deployed to Preview – petrinaut-docs September 2, 2026 23:57 View deployment

vercel Bot deployed to Preview – hash September 3, 2026 00:04 View deployment

Mark every spoken answer with icon provenance

5ea4ede

Co-authored-by: Cursor <cursoragent@cursor.com>

lunelson mentioned this pull request Sep 3, 2026

FE-1563: Establish Brunch elicitation skill boundaries #9468

Merged

4 tasks

vercel Bot deployed to Preview – petrinaut-docs September 3, 2026 09:50 View deployment

vercel Bot deployed to Preview – petrinaut September 3, 2026 09:51 View deployment

cursor Bot reviewed Sep 3, 2026

View reviewed changes

Comment thread libs/@hashintel/petrinaut/src/ui/views/Editor/panels/ai-assistant-panel.tsx

vercel Bot deployed to Preview – hash September 3, 2026 09:58 View deployment

Center Voice provenance icons

aa60205

Co-authored-by: Cursor <cursoragent@cursor.com>

vercel Bot deployed to Preview – petrinaut September 3, 2026 10:02 View deployment

vercel Bot deployed to Preview – petrinaut-docs September 3, 2026 10:03 View deployment

vercel Bot deployed to Preview – hash September 3, 2026 10:08 View deployment

Fix Voice handoff and provenance races

d13535d

Co-authored-by: Cursor <cursoragent@cursor.com>

vercel Bot deployed to Preview – petrinaut September 3, 2026 12:35 View deployment

vercel Bot deployed to Preview – petrinaut-docs September 3, 2026 12:35 View deployment

cursor Bot reviewed Sep 3, 2026

View reviewed changes

vercel Bot deployed to Preview – hash September 3, 2026 12:43 View deployment

This was referenced Sep 3, 2026

FE-1576: Ground Brunch interviews in the current Petrinaut net #9523

Open

FE-1580: Reconcile Voice turn behavior on the shared Brunch conversation #9531

Draft

lunelson mentioned this pull request Sep 4, 2026

FE-1574: Let Voice and the typed panel share one canonical Brunch conversation #9528

Open

5 tasks

Navigation Menu

Uh oh!

H-6800: Improve Petrinaut Voice turn-taking and answer provenance - #9512

H-6800: Improve Petrinaut Voice turn-taking and answer provenance#9512
kostandinang wants to merge 10 commits into
kah-6763-temporary-brunch-askfrom
kah-6800-improve-petrinaut-voice-turn-taking-and-answer-provenance

kostandinang commented Sep 2, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

cursor Bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

kostandinang commented Sep 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

kostandinang commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌟 What is the purpose of this PR?

🔗 Related links

🚫 Blocked by

🔍 What does this change?

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

📜 Does this require a change to the docs?

🕸️ Does this require a change to the Turbo Graph?

⚠️ Known issues

🐾 Next steps

🛡 What tests cover this?

❓ How to test this?

📹 Demo

Uh oh!

vercel Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

kostandinang commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Sep 3, 2026

Choose a reason for hiding this comment

Rollback restores stale voice IDs

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

1 participant

kostandinang commented Sep 2, 2026 •

edited

Loading

vercel Bot commented Sep 2, 2026 •

edited

Loading

cursor Bot commented Sep 2, 2026 •

edited

Loading

kostandinang commented Sep 2, 2026 •

edited

Loading